{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intent Resolution Evaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting Started\n",
    "This sample demonstrates to how to use intent resolution evaluator on agent data. The supported input formats include:\n",
    "- simple data such as strings;\n",
    "- user-agent conversations in the form of list of agent messages. \n",
    "\n",
    "Before you begin:\n",
    "```bash\n",
    "pip install azure-ai-projects azure-identity azure-ai-evaluation\n",
    "```\n",
    "Set these environment variables with your own values:\n",
    "1) **PROJECT_CONNECTION_STRING** - The project connection string, as found in the overview page of your Azure AI Foundry project.\n",
    "2) **MODEL_DEPLOYMENT_NAME** - The deployment name of the AI model, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
    "3) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
    "4) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
    "5) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation.\n",
    "6) **AZURE_SUBSCRIPTION_ID** - Azure Subscription Id of Azure AI Project\n",
    "7) **PROJECT_NAME** - Azure AI Project Name\n",
    "8) **RESOURCE_GROUP_NAME** - Azure AI Project Resource Group Name\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Intent Resolution evaluator measures how well an agent has identified and resolved the user intent.\n",
    "The scoring is on a 1-5 integer scale and is as follows:\n",
    "\n",
    "  - Score 1: Response completely unrelated to user intent\n",
    "  - Score 2: Response minimally relates to user intent\n",
    "  - Score 3: Response partially addresses the user intent but lacks complete details\n",
    "  - Score 4: Response addresses the user intent with moderate accuracy but has minor inaccuracies or omissions\n",
    "  - Score 5: Response directly addresses the user intent and fully resolves it\n",
    "\n",
    "The evaluation requires the following inputs:\n",
    "\n",
    "  - Query    : The user query. Either a string with a user request or a list of messages with previous requests from the user and responses from the assistant, potentially including a system message.\n",
    "  - Response : The response to be evaluated. Either a string or a message with the response from the agent to the last user query.\n",
    "\n",
    "There is a third optional parameter:\n",
    "  - ToolDefinitions : The list of tool definitions the agent can call. This may be useful for the evaluator to better assess if the right tool was called to resolve a given intent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Intent Resolution Evaluator\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azure.ai.evaluation import AzureOpenAIModelConfiguration\n",
    "from azure.identity import DefaultAzureCredential\n",
    "from azure.ai.evaluation import IntentResolutionEvaluator\n",
    "from pprint import pprint\n",
    "\n",
    "model_config = AzureOpenAIModelConfiguration(\n",
    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
    ")\n",
    "\n",
    "intent_resolution_evaluator = IntentResolutionEvaluator(model_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating query and response as string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Success example. Intent is identified and understood and the response correctly resolves user intent\n",
    "result = intent_resolution_evaluator(query=\"What are the opening hours of the Eiffel Tower?\",\n",
    "                                     response=\"Opening hours of the Eiffel Tower are 9:00 AM to 11:00 PM.\",\n",
    "                                    )\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Failure example. Even though intent is correctly identified, the response does not resolve the user intent\n",
    "result = intent_resolution_evaluator(query=\"What is the opening hours of the Eiffel Tower?\",\n",
    "                                     response=\"Please check the official website for the up-to-date information on Eiffel Tower opening hours.\",\n",
    "                                    )\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating query and response as strings with tool definition as single dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example with a single tool definition dict instead of a list\n",
    "query = \"What's the status of order #789?\"\n",
    "response = \"Order #789 has been shipped and will arrive on March 18, 2025.\"\n",
    "\n",
    "tool_definition_dict = {\n",
    "    \"name\": \"get_order\",\n",
    "    \"description\": \"Get the details of a specific order.\",\n",
    "    \"parameters\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\"order_id\": {\"type\": \"string\", \"description\": \"The order ID to get the details for.\"}},\n",
    "    },\n",
    "}\n",
    "\n",
    "result = intent_resolution_evaluator(\n",
    "    query=query,\n",
    "    response=response,\n",
    "    tool_definitions=tool_definition_dict,\n",
    ")\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating query and response as list of messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = [\n",
    "    {\"role\": \"system\", \"content\": \"You are a friendly and helpful customer service agent.\"},\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T06:14:20Z\",\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"Hi, I need help with the last 2 orders on my account #888. Could you please update me on their status?\",\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "]\n",
    "\n",
    "response = [\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T06:14:30Z\",\n",
    "        \"run_id\": \"0\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [{\"type\": \"text\", \"text\": \"Hello! Let me quickly look up your account details.\"}],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T06:14:35Z\",\n",
    "        \"run_id\": \"0\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"tool_call_20250310_001\",\n",
    "                \"name\": \"get_orders\",\n",
    "                \"arguments\": {\"account_number\": \"888\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T06:14:40Z\",\n",
    "        \"run_id\": \"0\",\n",
    "        \"tool_call_id\": \"tool_call_20250310_001\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": '[{ \"order_id\": \"123\" }, { \"order_id\": \"124\" }]'}],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T06:14:45Z\",\n",
    "        \"run_id\": \"0\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"Thanks for your patience. I see two orders on your account. Let me fetch the details for both.\",\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T06:14:50Z\",\n",
    "        \"run_id\": \"0\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"tool_call_20250310_002\",\n",
    "                \"name\": \"get_order\",\n",
    "                \"arguments\": {\"order_id\": \"123\"},\n",
    "            },\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"tool_call_20250310_003\",\n",
    "                \"name\": \"get_order\",\n",
    "                \"arguments\": {\"order_id\": \"124\"},\n",
    "            },\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T06:14:55Z\",\n",
    "        \"run_id\": \"0\",\n",
    "        \"tool_call_id\": \"tool_call_20250310_002\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_result\",\n",
    "                \"tool_result\": '{ \"order\": { \"id\": \"123\", \"status\": \"shipped\", \"delivery_date\": \"2025-03-15\" } }',\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T06:15:00Z\",\n",
    "        \"run_id\": \"0\",\n",
    "        \"tool_call_id\": \"tool_call_20250310_003\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_result\",\n",
    "                \"tool_result\": '{ \"order\": { \"id\": \"124\", \"status\": \"delayed\", \"expected_delivery\": \"2025-03-20\" } }',\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T06:15:05Z\",\n",
    "        \"run_id\": \"0\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"The order with ID 123 has been shipped and is expected to be delivered on March 15, 2025. However, the order with ID 124 is delayed and should now arrive by March 20, 2025. Is there anything else I can help you with?\",\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "]\n",
    "\n",
    "# please note that the tool definitions are not strictly required, and that some of the tools below are not used in the example above and that is ok.\n",
    "# if context length is a concern you can remove the unused tool definitions or even the tool definitions altogether as the impact to the intent resolution evaluation is usual minimal.\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"get_orders\",\n",
    "        \"description\": \"Get the list of orders for a given account number.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"account_number\": {\"type\": \"string\", \"description\": \"The account number to get the orders for.\"}\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"get_order\",\n",
    "        \"description\": \"Get the details of a specific order.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"order_id\": {\"type\": \"string\", \"description\": \"The order ID to get the details for.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"initiate_return\",\n",
    "        \"description\": \"Initiate the return process for an order.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"order_id\": {\"type\": \"string\", \"description\": \"The order ID for the return process.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"update_shipping_address\",\n",
    "        \"description\": \"Update the shipping address for a given account.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"account_number\": {\"type\": \"string\", \"description\": \"The account number to update.\"},\n",
    "                \"new_address\": {\"type\": \"string\", \"description\": \"The new shipping address.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "result = intent_resolution_evaluator(\n",
    "    query=query,\n",
    "    response=response,\n",
    "    tool_definitions=tool_definitions,\n",
    ")\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating an agent conversation loaded from disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from azure.ai.evaluation import AIAgentConverter\n",
    "\n",
    "def load_conversations(filename):\n",
    "    with open(filename, 'r') as file:\n",
    "        lines = file.readlines()\n",
    "        parsed_conversations = [json.loads(line) for line in lines]\n",
    "    print(f\"Loaded {len(parsed_conversations)} conversations from {filename}.\")\n",
    "    return parsed_conversations\n",
    "\n",
    "conversations_filename = r'sample_synthetic_conversations.jsonl'\n",
    "\n",
    "#this loads 90 conversations from the file sample_synthetic_conversations.jsonl\n",
    "sample_conversations = load_conversations(conversations_filename)\n",
    "\n",
    "#get the first conversation from the loaded conversations\n",
    "conversation = sample_conversations[10]\n",
    "\n",
    "run_ids = AIAgentConverter.run_ids_from_conversation(conversation)\n",
    "print(f\"Run IDs in conversation: {run_ids}\")\n",
    "run_id = str(run_ids[0]) # convert run_id to string in case it is some other type, e.g. an int\n",
    "converted_conv = AIAgentConverter.convert_from_conversation(conversation, run_id)\n",
    "# Extract the query and response from the conversation\n",
    "query = converted_conv['query']\n",
    "response = converted_conv['response']\n",
    "tool_definitions = converted_conv['tool_definitions']\n",
    "\n",
    "print(f\"Run ID: {run_id}\")\n",
    "print(f\"Query: {query}\")\n",
    "print(f\"Response: {response}\")\n",
    "print(f\"Tool Definitions: {tool_definitions}\")\n",
    "\n",
    "result = intent_resolution_evaluator(query = query, response = response, tool_definitions = tool_definitions)\n",
    "print(f\"Evaluation result\")\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Putting it all together and evaluate an entire conversation run by run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_conversation_run(conversation : dict, run_id : str, verbose=False):\n",
    "    converted_conv = AIAgentConverter._convert_from_conversation(conversation, str(run_id))\n",
    "    # Extract the query and response from the conversation\n",
    "    query = converted_conv['query']\n",
    "    response = converted_conv['response']\n",
    "    tool_definitions = converted_conv['tool_definitions']\n",
    "    \n",
    "    if verbose:\n",
    "        print(f\"*********************************************\")\n",
    "        print(f\"Evaluating conversation run with ID: {run_id}\")\n",
    "        print(f\"Run ID: {run_id}\")\n",
    "        print(f\"Query: {query}\")\n",
    "        print(f\"Response: {response}\")\n",
    "        print(f\"Tool Definitions: {tool_definitions}\")\n",
    "\n",
    "    # Evaluate the query and response using the intent resolution evaluator\n",
    "    evaluation_result = intent_resolution_evaluator(query = query, response = response, tool_definitions = tool_definitions)\n",
    "    if verbose:\n",
    "        print(f\"Evaluation Result:\")\n",
    "        pprint(evaluation_result)\n",
    "\n",
    "    return evaluation_result\n",
    "\n",
    "def evaluate_conversation(conversation, verbose=False):\n",
    "    run_ids = AIAgentConverter._run_ids_from_conversation(conversation)\n",
    "    print(f\"Runs available in conversation: {run_ids}\")\n",
    "    results = []\n",
    "    for run_id in run_ids:\n",
    "        result = evaluate_conversation_run(conversation, str(run_id), verbose)\n",
    "        results.append(result)\n",
    "    return results\n",
    "\n",
    "sample_conversation = sample_conversations[20]\n",
    "evaluate_conversation(sample_conversation, verbose=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_agent_evaluator_prp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
